Information-theoretics Based Error-metrics for Gradient Descent Learning in Neutral Networks

Conventionally, square error (SE) and/or relative ent ropy (RE) error functions defined over a training set are adop ted towa rds optimization of gradient descent learnings in neur al networks. As an alternative, a set of divergence (or dist ance) measur es can be specified in the inform ationtheoretic plane that funct iona lly have pragmatic values similar to (or improved upon) The SE or RE metrics. KullbackLeibler (KL) , Jensen (J ), and Jensen-Sh ann on (JS) meas ures are suggested as possible inform ation-theoretic error-met ric candidates that are defined and derived explicit ly. Both convent ional SE / RE measures , as well as the prop osed inform ation-theoreti c error-met rics, are applied to t rain a multilayer perceptron topology. This is done in order to elucida te t heir relative efficacy in deciding the perform ance of the network as evidenced from the convergence rat es and tr aining t imes involved. Pertinent simulation results are present ed and discussed.